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(57) ABSTRACT 

Performance measurement and analysis of local exchange 
carrier interconnections. Performance measurements can be 
used to establish that an incumbent local exchange carrier 
("ILEC") is providing interconnections to one or more 
competitive local exchange carriers ("CLECs") that are at 
least equal in quality to the interconnections provided to 
itself. Performance measurements are defined to measure the 
timeliness, accuracy and availability of the interconnections 
provided to the CLECs. A performance measurement is 
based upon performance data that us collected from one of 
the ILECs processes (preordering, ordering, provisioning, 
collocation, billing, maintenance and repair, emergency 911, 
operator service/directory assistance and trunk blockage). A 
performance report is defined to specify the types of per- 
formance measurements and dimensions that are included, 
as well as the reporting period. 

15 Claims, 4 Drawing Sheets 
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METHOD FOR PERFORMANCE 
MEASUREMENT AND ANALYSIS OF LOCAL 
EXCHANGE CARRIER 
INTERCONNECTIONS 

RELATED APPLICATION 

This application is a continuation of U.S. patent applica- 
tion Sen No. 09/474,356 filed Dec. 29, 1999 now U.S. Pat. 
No. 6,480,749 (hereinafter "parent application") which is 
incorporated herein by recreance. 

The parent application was granted priority to U.S. Pro- 
visional Patent Application Ser. No. 60/164,682 entitled "A 
System and Method for Collecting and Analyzing Perfor- 
mance Measurements for Telecommunications Systems", 
filed Nov. 10, 1999 which is incorporated herein by refer- 
ence. The present application, the parent application and the 
U.S. provisional patent application are commonly assigned 
to Bell South Intellectual Property Corporation. 

TECHNICAL FIELD 

This invention relates in general to the collection and 
analysis of data, and more particularly to collecting data to 
determine a performance measurement that measures an 
interconnection provided by a local exchange carrier. 

BACKGROUND OF THE INVENTION 

A local exchange carrier ("LEC") utilizes many different 
processes in order to serve its customers. For example, one 
or more processes typically supports customer ordering. 
Customer ordering includes the initiation of new service or 
the modification of existing service. Other processes support 
maintenance and repair and billing. 

At one time, a local telephone market was served by a 
single LEC. However, local telephone markets now are open 
to competition from competitive local exchange carriers 
("CLEC"). The existing or incumbent LEC ("ILEC") is 
required to offer quality interconnection services, resale of 
capacity at wholesale rates, dialing parity, number portabil- 
ity and unbundled access to its networks to the CLECs. 
Although the ILEC is required to satisfy these requirements, 
there is no established method for measuring the ILEC's 
compliance. For example, the ILEC has a duty not to 
prohibit, and not to impose unreasonable or discriminatory 
conditions or limitations on, the resale of its telecommuni- 
cations services. However, there is no well-defined method 
or measurement to insure that this duty is met. 

The telecommunications industry and the FCC issued a 
notice of proposed rule making ("NPRM") that set forth 
some model performance measurements for measuring an 
ILEC's compliance. However, no rules have been promul- 
gated. In addition, some states or public service commis- 
sions have attempted to define performance measurements, 
but either the performance measurements have not been 
enacted or the performance measurements are not well- 
defined. Therefore, there is a need to define performance 
measurements that establish that an ILEC has satisfied the 
requirements to provide equivalent service to a CLEC. 

Once the performance measurements have been defined, 
then the ILEC must also determine how to collect the 
required data and present it in a useable format. Because the 
requirements to provide equivalent service to a CLEC cover 
a wide range of services, the ILEC must collect data from the 
various processes that it uses. These processes can be 
located on systems that are physically separate from one 
another. In addition, the processes can use data formats that 
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are incompatible. In order to determine the performance 
measurements, the ILEC must identify the data that needs to 
be collected. Once the data is collected, the system must then 
normalize the data or transform the data into a common 

5 format so that data from multiple systems can be used to 
determine the performance measurements. Therefore, there 
is a need for a method of collecting data from a variety of 
systems or processes that may be incompatible with one 
another, normalizing the data and using the data to determine 

io performance measurements. 

Although it may be possible for the performance mea- 
surements to be determined manually, the manual collection 
and analysis of data greatly limits the number of measure- 
ments that can be taken. In addition, if the system is 

15 manually intensive, then it is difficult to alter the types of 
data that are collected or to alter the types of reports or other 
analysis that is generated from the data. Thus, the method for 
collecting, normalizing and analyzing the data should be 
automated. In addition, the method should be flexible so that 

20 the types of data collected and the reports generated can be 
easily modified. 

SUMMARY OF THE INVENTION 

The present invention meets the needs described above by 

25 providing a method for defining, analyzing and reporting 
performance measurements. The performance measure- 
ments can be used to establish that an incumbent local 
exchange carrier ("ILEC") is providing interconnections to 
one or more competitive local exchange carriers ("CLECs") 

30 that are at least equal in quality to the interconnections 
provided to itself. 

Typically, performance measurements are defined to mea- 
sure such things as timeliness, accuracy and availability. The 
performance measurements can be used to compare the 

35 services provided by the ILEC to the CLEC's to the services 
provided by the ILEC to itself. An ILEC utilizes many 
different processes to serve its customers. Exemplary pro- 
cesses include preordering, ordering, provisioning, 
collocation, billing, maintenance and repair, emergency 911, 

40 operator service/directory assistance and trunk blockage. 
A performance measurement is based upon a calculation 
that uses performance data collected from the processes. 
When the performance data is collected, the data is identified 
with one or more dimensions, such as the geography, entity, 

45 product and time dimensions. A dimension defines how a 
performance measurement is reported. 

The method for defining, determining and reporting a 
performance measurement includes the steps of defining the 
performance measurement and defining the dimensions for 

50 the performance measurement. The method also includes 
defining the performance data needed to determine the 
performance measurement. In some instances, the perfor- 
mance data needed to determine a performance measure- 
ment is used for other purposes and is thus, available from 

55 the process. However, in other instances, the performance 
data is created or collected especially for the performance 
measurement. For example, a performance measurement 
based upon timeliness may require that the process use 
timestamps when such timestamps were not previously 

60 used. If timestamps are required, then the process associates 
a time-stamp with certain events in order to measure an 
interval or response time. The method also includes defining 
the performance reports to specify the types of performance 
measurements and dimensions that are included, as well as 

65 the reporting period. Preferably, the definition of the report 
can be easily modified to adapt to changes in the 
requirements, processes or user requests. 
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The performance data needed to determine the perfor- interconnections provided to the ILEC itself. Briefly 

mance measurements is obtained and is used to determine described, the performance measurements measure the 

the performance measurement. The performance measure- timeliness, accuracy and effectiveness of the interconnec- 

ment can be determined using a combination of dimensions. tions provided by the ILEC. The performance measurements 

For example, a performance measurement can be calculated 5 are determined using performance data that is collected from 

for a particular CLEC in a particular geographic area for a systems associated with the ILEC. Typically, the systems 

particular time period. Once the performance measurement correspond to the processes used by the ILEC to support the 

is determined it can be included in the performance report. interconnections, such as ordering and billing. The collected 

A performance measurement and analysis platform performance data is normalized and the data is used to 
("PMAP") system supports the collection of performance ™ determine the performance measurements. The performance 
data, the determination of the performance measurements measurements are reported using a variety of formats. The 
and the generation of the performance reports. The PMAP performance measurements for a particular CLEC can be 
system includes source systems, a staging database, a nor- reported or performance measurements for all CLECs can 
malized operational data store, a dimensional data store be reported. The performance measurements for one or more 
database and a user interface. A number of source systems 15 CLECs can also be compared to the performance measure- 
provide data to the PMAP system. Typically, the source ment s for the ILEC. 
systems correspond to the processes used by the ILEC. 

Once the data is collected from the source systems, the 

data is loaded into a staging database and the data is filtered Typically, a local exchange carrier ("LEC) utilizes many 

and normalized. The normalized operational data store is 20 different processes to serve its customers. The exemplary 

used to validate the data against business rules and data processes illustrated in FIG. 1 include preordering 100, 

relationships and transform the data to conform to the PMAP ordering 102, provisioning 104, collocation 106, billing 108, 

data model. The dimensional data store database includes maintenance and repair 110, emergency 911 112, operator 

performance measurements which include aggregate and service/directory assistance 114, and trunk blockage 116. As 

summary data. The PMAP system provides a variety of 25 will be apparent to those skilled in the art, different LECs 

reporting capabilities. The reports include aggregate and may have different processes than those illustrated by FIG. 

CLEC-specific reports, state and regional reports, and 1. Some LECs may combine one or more of the processes 

reports directed to the different processes or subject areas. illustrated in FIG. 1 into a single process or may divide a 

The PMAP system also creates raw data files that contain sin g* e process illustrated in FIG. 1 into multiple processes, 

detailed information about specific local service requests, FIG - 1 also illustrates customer events 118, The customer 

service orders trouble tickets and other items that are typi- events interface directly with some of the processes. In FIG. 

cally reported. Typically, the raw data is used to recreate tDe customer events 118 interface with the emergency 911 

performance reports or to enable a user to create a custom billing 108, operator service/directory assistance 114, 

report. A user can download raw data files, import raw data 35 and trunk blockage 116 processes. 

files, import raw data files into a program, such as a The preordering process 100 is directed to activities that 

spreadsheet program, or manipulate the raw data to create a occur prior to the submittal of a local service request by an 

measurement in any of the performance reports. LEC. These activities include verifying the customer's street 

These and other aspects, features and advantages of the address, determining available products and services, esti- 

present invention may be more clearly understood and 40 matin g tne service interval and reserving a telephone num- 

appreciated from a review of the following detailed descrip- ber - Data validation is performed to ensure that the local 

tion of the disclosed embodiments and by reference to the service request is complete and accurate. Performance mea- 

appended drawings and claims. surements directed to the preordering process include the 

availability of the preordering process and the response 

BRIEF DESCRIPTION OF THE DRAWINGS 45 time. 

FIG. 1 is a block diagram illustrating exemplary , ™ e orderio S P roccss ™ 2 u wheE l an ^ enters a 

processes, in accordance with an embodiment of the present loc ^ u request * nd u cnd ? when the , P~ receives 

invention confirmation that an order has been created in the system. 

. . . , , .„ , Orders may be submitted electronically or via facsimile, 

FIG. 2 is a block diagram illustrating exemplary telephone or e-mail. Performance measurements directed to 

dimensions, m accordance with an embodiment of the the ordering proce ss include availability of order progress 

present information. information. 

FIG. 3 is a flow chart illustrating the definition of a Xhe prov isioning process 104 includes facilities 

performance measurement, in accordance with an embodi- assignment, software changes, service design, issuance of 

ment of the present information. 55 technician work orders and activation procedures. The pro- 

FIG. 4 is a block diagram illustrating the processing of visioning process ends when a billing record is created for 

performance data, in accordance with an embodiment of the the new account, or the billing record is updated if the order 

present information. - is being provisioned for a change order._Performance mea 



DETAILED DESCRIPTION OF THE 
INVENTION 



surements directed to the provisioning process include the 
60 timeliness of service delivery and the accuracy of. the 
services provided to the LEC. 
The present invention is directed to a method for defining, The collocation process 106 includes activities related to 
analyzing and reporting performance measurements. The placing customer-owned equipment in the ILECs central 
performance measurements can be used to establish that an office for interconnection to the ILECs tariffed services and 
incumbent local exchange carrier ("ILEC") is providing 65 unbundled network elements. Performance measurements 
interconnections to one or more competitive local exchange directed to the provisioning process include the timeliness of 
carriers. ("CLECs") that are at least equal in quality to the the interconnection. 
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The billing process 108 includes the activities associated 
with accumulated usage data and determining the charges to 
be billed to a customer's account. Performance measure- 
ments directed to the billing process include the accuracy 
and the timeliness of customer invoices. 

The maintenance and repair process 110 includes activi- 
ties directed to responding to a customer's maintenance and 
repair needs. The maintenance and repair process begins 
when a customer reports a service problem. A trouble ticket 
is entered to document the problem. The equipment and 
facilities are tested to locate the source of the trouble and 
once the problem is repaired, the customer is notified and the 
trouble ticket is closed. Performance measurements directed 
to the maintenance and repair process include the timeliness 
o f the repair and the rate of repeat problems. 

The emergency 911 process 112 includes activities that 
support emergency 911 service. In one embodiment, the 
emergency 911 process includes database updates to a third 
party emergency 911 vendor to insure that the vendor has the 
most up-to-date information for providing emergency ser- 
vice to residents and businesses. In other embodiments, the 
emergency 911 process is directed to activities that include 
the actual provision of emergency 911 service. Performance 
measurements directed to the emergency 911 process 
include the timeliness and accuracy of the service. 

The operator service/directory assistance process 114 
includes activities required to provide additional services to 
the customer, such as directory inquires to retrieve telephone 
numbers. Performance measurements directed to the opera- 
tor services process include the timeliness of the service. 

The trunk blockage process 116 includes the collection of 
traffic performance data on the trunk groups in the network. 
Performance measurements directed to the trunk blockage 
process include the number of attempted calls and the 
number of blocked calls. 

Definition of Performance Measurements 

Performance measurements are defined to demonstrate 
that an ILEC is providing interconnections to CLECs that 
are at least equal in quality to those provided by the ILEC 
to itself. The performance measurements typically are used 
to compare the services provided by the. ILEC to the 
CLECs to the services provided by the ILEC to itself. In 
some instances a comparison is made between the ILEC and 
all other CLECs. In other instances a comparison is made 
between the ILEC and a particular CLEC. 
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Performance measurements are defined to measure such 
things as timeliness, accuracy and availability. In the exem- 
plary embodiment described herein, the performance mea- 
surements are defined to measure activities associated with 

5 the different processes described in the preceding section. 
Typically, a performance measurement is defined as a total, 
percentage, interval or accuracy measurement. A perfor- 
mance measurement that measures a total is defined to be the 
sum of a number of events or occurrences. For example, a 

10 flow through error analysis performance measurement asso- 
ciated with the ordering process is defined as the sum of 
errors by type. 

A performance measurement that measures a percentage 
is defined to be an actual number divided by a total or 
scheduled number and multiplied by 100. For example, the 
missed repair appointments performance measurement asso- 
ciated with the maintenance and repair process is defined as 
follows: 

20 Percentage of missed repair appointments=2(count. of 
customer troubles not cleared by the quoted commit- 
ment date and time)/2(total trouble reports closed in 
reporting period)xl00 
A performance measurement that measures an interval is 
25 defined to be an actual time interval for an event divided by 
the total number of events. For example, the average 
completion interval performance measurement associated 
with the provisioning process is defined as follows: 

Average Completion Interval=2[(completion date and 
30 time) — (order issue date and time)]/2(total orders com- 
pleted in reporting period) 
A performance measurement that measures accuracy is 
defined to be the percentage of correct events to total events. 
For example, the invoice accuracy performance measure- 
35 ment associated with the billing process is defined as fol- 
lows: 

Invoice Accuracy=(total billed revenues) — (billing 

related adjustments)/(total billed revenues) 
Exemplary performance measurements associated with 
40 the processes are summarized in the following tables. Each 
table corresponds to one of the previously described pro- 
cesses and includes the name of the performance measure- 
ments and the definition of each performance measurement. 
In addition, each table includes the dimensions and perfor- 
45 mance data associated with the process. The dimensions and 
the performance data can be associated with more than one 
of the performance measurements in the table. 



TABLE 1 



PRE-ORDERING PROCESS 



Performance Measurements 


Performance Definition 


Dimensions 


Performance Data 


Average OSS Response 


2(Date and Tune of Legacy Response) - 


Tune 


Number of Legacy 


Interval 


(Date and Tune of Request to Legacy)]/ 




Requests 




(Number of Legacy Requests During the 








Reporting Period) x 100 






OSS Interface Availability 


(Functional Availability)/ 


Geography 


Summation of Response 




(Scheduled Availability) x 100 




Interval 
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TABLE 2 



ORDERING PROCESS 

Performance Measurements Performance Definition Dimensions Performance Data 



Percent Flow Through Service 
Requests 



Percent Rejected Service 



Reject Interval 



Firm Order Confirmation 
Timeliness 



Speed of Answer in Ordering 
Center 



EfTotal number of valid service requests 
that flow through to the ILEC OSS)/(Total 
number of valid service requests 
delivered to the ILEC OSS) x 100 
(Total Number of Rejected Service 
Requests)/Total Number of Service 
Requests Received) x 100 
2[(Date and Time of Service Request 
Rejection) - (Date and Tune of Service 
Request Receipt)]/(Number of Service 
Requests Rejected in Reporting Period) 
X[(Date and Tune of Firm Order 
Confirmation) - (Date and Time of Service 
Request Receipt^Number of Service 
Requests Confirmed in Reporting Period) 
(Total Time in second to reach the Local 
Carrier Service Center)/(Total Number of 
Calls) in the Reporting Period 



Flow Through Error Analysis 2 of errors by type 



Entity Total Number Rejected 

Service Requests 



Time Total Number Service 

Requests Received 

Geography Date and Time of Service 

Request rejection 



Date and Time of Service 
Request receipt 



Date and Time of Firm 
OTder Confirmation 

Number Service Requests 
confirmed (in reporting period) 



Class of Service/ 
Product 



TABLE 3 



PROVISIONING 



Performance Measurements Performance Definition 



Dimensions 



Performance Data 



Average Completion Interval 



Order Completion Interval 
Distribution 

Mean Held Order Interval 



Held Order Distribution 
Interval 

Average Jeopardy Notice 
Interval 



Percentage of Orders Given 
Jeopardy Notices 

Percent Missed Installation 
Appointments 



Percent Provisioning Troubles 
within 30 days 



Coordinated Customer 
Conversions 



Average Completion Notice 
Interval 



^[(Completion Date and time) - (Order 
Issue Date and Time)]/2:(Count of Orders 
Completed in Reporting Period) 
^(Service Orders Completed in "X" 
days)/Total Service Orders Completed 
in Reporting Period) x 100 
^(Reporting Period Close Date - 
Committed Order Due Date)/Number of 
Orders Pending and Past the Committed 
Due Date) for all orders pending and 
past the committed due date 
(Number of Orders Held for ^ "X" days)/ 
(Total number of orders pending but not 
completed) x 100 
^(Reporting Period Close Date - 
Committed Order Due Date)/(Number of 
Orders Pending and Past the Committed 
Due Date) for all orders pending and 
past the committed due date 
X[(Number of Orders Given Jeopardy 
Notices in Reporting Period)/(Number of 
Orders Committed in Reporting Period) 
^(Number of Orders Not Complete by 
Committed Due Date in Reporting Period)/ 
(Number of Orders Completed in 
Reporting Period) x 100 
ZfTrouble reports on all completed orders ^ 
30 days following service order(s) 
completion)/(All Service Orders in a 
completed in the report calendar month) x 100 
^[(Completion Date and Time for Cross 
Connection of an Unbundled Loop) - 
(Disconnection Date and Tune of an 
Unbundled Loop)J/Total Number of 
Unbundled Loop Items for the reporting period. 
Z(Date and Time of Notice of Completion) - 
(Date and Time of Work Completion)/ 
(Number of Orders Completed in 
Reporting Period) 



Entity 



Tune 



Geography 



Class of Service/ 
Product 



Total Service Orders 
Completed 

Service Order Completion 
Date and Time 

Service Order Issue Date 
and Time 



Service Orders completed 
in "X" Days 

Committed Order Due 
Date 



Number of Service Orders 
Pending and Past The 
Committed Due Date 
Number of Service Orders 
Held for >=90 Days 

Number of Service Orders 
Held for >-15 Days 



Total Number of Service 
Orders Pending But Not 
Completed 



Number of Service Orders 
missed in Reporting Period 
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TABLE 4 



COLLOCATION 



Performance Measurements Performance Definition 



Dimensions Performance Data 



Average Response Time ^(Request Response Date) - (Request Entity 
Submission Date)/Count of Responses 
Returned within Reporting Period 

Average Arrangement Time Z(Date Collocation Arrangement is Time 
Complete) - (Date Order for Collocation 
Arrangement Submitted)/Total Number of 
Collocation Arrangements Completed during 
Reporting Period 

Percent of Due Dates 2(Numbcr of Orders not completed within Geography 

Missed ILEC Committed Due Date during Reporting 

Period)/Number of Orders Completed in 

Reporting Period) x 100 



Request Response Date 
& Time 

Request Submission Date 
& Time 



Count of Requests 
Submitted 



TABLE 5 

BILLING 



Performance Measurements Performance Definition Dimensions Performance Data 



Invoice Accuracy 


(Total Billed Revenues during current month) - 
(Billing Related Adjustments during current 
month)/Total Billed Revenues during 


Entity 


Total Local Services billed 
Revenues 




current month x 100 




Total Adjustment 
Revenues 


Invoice Timeliness 


£(total number of usage records sent within 
six (6) calendar days from initial recording/ 
rcceipt)/Z(Total number of usage records 
sent) x 100 


Time 


Usage Data Delivery 


2 (total number of usage records sent within six 


Geography 


Summation of Time to 


Completeness 


(6) calendar days from initial recording/receipt)/ 
£(Total number of usage records sent) x 100 




Transmit Invoices 


Usage Data Delivery 


2(Total number of usage records sent within 




Total No. of Invoices 


Timeliness 


six calendar days from initial recording/receipt)/ 
E(Total number of usage records sent) x 100 






Usage Data Delivery 


I[(Total number of usage data packs sent 


Class of Service/ 


Number of Usage Data 


Accuracy 


during current month) - (Total number of usage 
data packs requiring retransmission during 
current month) ^Total number of usage data 
packs sent during current month) x 100 


Product 


Packs Sent 



TABLE 6 

MAINTENACE AND REPAIR 



Performance Measurements Performance Definition 



Dimensions 



Performance Data 



OSS Interface Availability 

Average OSS Response 
Interval 



Average Answer Time - 
Repair 



Missed Repair 
Appointments 

Customer Trouble Report 
Rate 



Maintenance Average 
Duration 

Out of Service ("OOS") 
>24 Hours 



(Actual System Functional Availability)/ 
(Actual planned System Availability) x 100 
(Query Response Date and Time for Category 
"X") - (Query Request Date and Tune for 
Category "X'^/(Number of Queries Submitted 
in the Reporting Period) where, "X" is 0-4, 
^4 to 10, ^10, ^30 seconds 
(Time ILEC Repair Attendant Answers Call) 
- (Time of entry into queue until ACD 
Selection)/(Total number of calls by 
reporting period 

2(Count of Customer Troubles Not Cleared by 
the Quoted Commitment Date and Time)/Z(Total 
Trouble reports closed in Reporting Period) x 100 
(Count of Initial and Repeated Trouble Reports 
in the Current Period)/(Number of Service 
Access Lines in Service at End of the 
Report Period) x 100 

£(Date and Time of Service Restoration) - 
(Date and Time Trouble Ticket was Opened)/ 
E(Total Closed Troubles in the Reporting Period 
(Total Troubles COS >24 Hours)/Total OOS 
Troubles in Reporting Period) x 100 



Entity 
Tune 



Geography 



Interval 
Distribution 



Class of Service/ 
Product 



Percent Repeat Troubles (Count of Customer Troubles where more than 



Total time in seconds for 
ILEC Repair Center Response 
Total Number of Calls 
Received 



Count of Customer 
Troubles Not Resolved 
by the Quoted 
Resolution Time and Date 
Count of Customer 
Trouble Tickets Closed 

Count of Repeated 
Trouble Reports in the 
Current Period 

Number of Service 
Access Lines in Service 
at End of the Report Period 
Total Duration Time from the 
Receipt to the Clearing of 
Trouble Reports 
Total Out of Service 
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TABLE 6-continued 



Performance Measurements 


MAINTENACE AND REPAIR 
Performance Definition Dimensions 


Performance Data 


within 30 days 


one trouble report was logged for the same 


Troubles 




service tine within a continuous 30 days)/ 






(Total Trouble Reports Closed in Reporting 






Period) x 100 




TABLE 7 



EMERGENCY 911 



Performance Measurements Performance Definition 



Dimensions Performance Data 



Timeliness 



Accuracy 



Mean Interval 



£ (Number of batch orders processed within 
24 hours + Total number of batch orders 
submitted) x 100 

2 (Number of record individual updates 
processed with no errors + Total number of 
individual record updates) x 100 
2(Date and time of batch order completion - 
Date and time of batch order submission)/ 
(Number of batch orders completed) 



Time Number of Confirmed 

Orders 

Geography Number of Orders missed 
in Reporting Period 

Total Number of SOIR 
orders for E911 Updates 



TABLE 8 





OPERATOR SERVICES 






Performance Measurements 


Performance Definition 


Dimensions 


Performance Data 


Average Speed to Answer 


(Total call waiting seconds)/Tbtal calls served) 


Time 


Call Waiting Seconds 


Percent Answered within 


(Total number of calls answered within X 


Geography 


Number of Calls served 


"X" seconds 


seconds)/Tbtal calls served) x 100 





Note: 

In some embodiments the operator services performance measurements are provided by the operator services process. 
No raw data is provided. 



TABLE 9 





TRUNK BLOCKAGE 




Performance 
Measurements 


Performance 
Definition 


Dimensions 


Performance Data 


Trunk Group 
Service Report 


(Total number of 
blocked cal1s)/fTotal 
number of attempted 
calls) x 100 


Entity 


Number of Trunk 
Groups Measured 



Dimensions 

A performance measurement is based upon performance 
data collected from the processes. When the performance 
data is collected, the data is identified with a-particular 
dimension. A dimension defines how a performance mea- 
-surement.based upon the data is reported. In the embodiment 
illustrated by FIG. 2, the primary dimensions include geog- 
raphy 200, entity 210, product 220 and time 230. 

The geography dimension permits performance measure- 
ments to be calculated based on specific geographic criteria. 
The geographic criteria shown in FIG. 2 are region 202, state 
204, MSA (market service area or metropolitan statistical 
area) 206 and wire center (or switching center) 208. The 
entity dimension permits performance measurements to be 
calculated based on a set of LEC's or a particular LEC. The 
entities shown in FIG. 2 include aggregate 212 (all CLECs 



and the ILEC), CLEC 214 (all CLECs), ILEC 216 

40 (incumbent LEC), CLEC company 218 (a particular CLEC) 
and CLEC Identifiers 219. 

The product dimension permits performance measure- 
ments to be calculated based upon specific products or 
services. The products shown in FIG. 2 include LA Product 

45 222, GA Product 224 and product class 226. The time 
dimension permits performance measurements to be calcu- 
lated based upon certain time intervals. The time dimensions 
shown in FIG. 2 include year 232, quarter 234 and month 
236. Multiple dimensions can be used to report performance 

50 measurements. For example, a report can be based on a 
particular region, CLEC and year. As will be apparent to 
those skilled in the art, other embodiments can include 
alternative or additional dimensions. 

55 As illustrated in FIG. 2, a dimension has multiple layers. 
If a performance report uses a particular layer of a dimension 
a user can obtain additional detail by accessing a different 
layer. For example, to obtain additional detail for a report 
that uses the "year" layer of the time dimension, the user can 

6Q access the "quarter" layer or the "month" layer. 

Method for Defining, Determining and Reporting 
Performance Measurement 

Exemplary steps for defining, determining and reporting 
65 a performance measurement are shown in FIG. 3. In step 
302, the performance measurement is defined. Defining the 
performance measurement includes defining a quantifiable 
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measure of an interconnection provided by the ILEC. The provide data to the PMAP system. Typically, the source 

performance measurements are typically associated with systems correspond to the processes previously described, 

one of the processes of the ILEC. For example, to define a The source systems can be existing legacy systems and can 

performance measurement for the maintenance and repair use data formats that are not compatible with one another, 

process, a performance measurement could be defined to 5 i n the exemplary embodiment illustrated by FIG. 4, the 

measure trouble reports not cleared by a committed date and LEO system 400a is a local exchange ordering system, the 

time (missed repair appointments measurement, Table 6). LON system 400fc is a local order number system, and the 

The preceding tables include exemplary definitions of per- EXACT system 400c is an exchange access control and 

formance measurements. In step 304, the dimensions for the tracking system. The LEO, LON and EXACT systems all 

performance measurement are defined. For example, the 10 correspond to the ordering process. The SOCS system 400d 

dimensions for the missed repair appointments measurement is a service order and control system and corresponds to the 

includes the entity, geography, product and time dimension. provisioning process. The WFA 400e system is a work force 

In step 306, the performance data needed to determine the administration system and corresponds to the maintenance 

performance measurement are defined. The performance and repair process. 

data are defined by considering the definition of the perfor- 15 -p^ CRIS system 400/ j s a customer record information 

mance measurement and the dimensions for the performance system and corresponds to the billing process. The LMOS 

measurement. For example, the performance data needed to system is a line maintenance operation system and corre- 

determine the missed repair appointments performance mea- sponds to the maintenance and repair process. The TIRKS 

surement are the count of customer troubles not cleared by system 300/t is a trunk integrated record keeping system and 

the quoted commitment date and time and the total trouble 20 corresponds to the trunk blockage process. The SOIR system 

reports closed in the reporting period. The performance data 400n is a service order information system and corresponds 

needed to determine the missed repair appointments perfor- to the emergency 911 process. As will be apparent to those 

mance measurement includes data associated with the entity, skilled in the art, alternative or additional source systems 

geography, product and time dimensions. ma y be included. In addition, a single system may provide 

In some instances, the performance data needed to deter- 25 data for multiple processes; 

mine a performance measurement is used for other purposes The data is collected from the source systems by access- 

and is thus, available from the process. However, in other ing existing databases or by retrieving the data manually, 

instances, the performance data is created or collected The manual retrieval of information can include receiving 

especially for the performance measurement. For example, data via facsimile, or e-mail and may require human inter- 

a performance measurement based upon timeliness may 30 vention to enter the data into the PMAP system 401. 

require that the process use timestamps when such times- Once the data is collected from the source systems 400a, 

tamps were not previously used. If timestamps are required, 400fc . . . 400/2 the data is loaded into a staging database 402. 

then the process associates a timestamp with certain events when the data is received by the staging database 402, the 

in order to measure an interval or response time. ^ data ^ not norm ahzed. Once in the staging database, the data 

In step 308, the performance reports are denned. The is filtered and normalized. Data errors are captured and 

performance reports are defined to specify the types of handled in the staging database. 

performance measurements and dimensions and the report- The NODS database 404 is a normalized operational data 
ing period. The reports can be defined by the ILEC, a CLEC store. In NODS, the data is validated against the business 
that is accessing the system or another entity. For example, 4Q rules and data relationships and transformed to conform to 
if the PSC (public service commission) requires a specific the PMAP data model. Between the NODS database and the 
type of report or a report that includes specific information, DDS database, the data undergoes an aggregation process, 
then the report can be defined to meet those requirements. ^ DDS database 406 is the dimensional data store 
Preferably, the definition of the report can be easily modified database. The DDS database includes performance measure- 
so that the report can adapt to changes in the requirements 45 ments which mclude aggregate and summary data. If access 
or processes or user requests. t0 detailed information is required, then the data must be 
In step 310, the performance data needed to determine the accessed by drilling down into the measurement data by 
performance measurements are obtained; The performance accessing the NODS database 404 or the raw data files 412 
data can be obtained from a number of different source as described below. 

systems associated with the different processes. Additional 50 The PMAP system creates raw data files 412 that contain 

details about the source systems are provided in the preced- detailed information about specific local service requests, 

ing section. In step 312, the performance data is used to service orders trouble tickets and other items that are typi- 

determine the performance measurement. A performance cally reported. Typically, the raw data is used to recreate 

measurement can be determined using a combination of performance reports or to enable a user to create a custom 

dimensions. For example, the percentage of missed repair 55 report. A user can download raw data files, import raw data 

appointments can be calculated for a particular CLEC in a files, import raw data files into a program, such as a 

particular geographic area for a particular quarter. In step spreadsheet program or manipulate the raw data to create a 

314 the performance measurement is included in the per- measurement in any of the performance reports, 

formance report. The performance report can be a written The PMAP system 401 provides a variety of reporting 

report or an on-line report. 60 capabilities. The reports 410 include aggregate and CLEC- 

Performance Measurement and Analysis Platform ™ 6 rCgi ° nal IC J? rt5 > 7*°** 

Svstem directed to the different processes or subject areas. In one 

embodiment, the reports can be accessed via a network, such 

FIG. 4 illustrates the logical layers of data that exist in the as the Internet. Typically, a user is provided with a user ID 

performance measurement and analysis platform ("PMAP") 65 and a password in order to access the reports (and the raw 

system 401 to support the required measurement and analy- data). Auser generally is permitted to access reports and data 

sis. A number of source systems 400a, 400f> . . . 400/2 related to all CLECs and the CLEC associated with the user, 
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but generally is not permitted to access reports and data 
related to another CLEC. A user associated with the incum- 
bent LEC can be provided with broader access to the data. 
As will be apparent to those skilled in the art, other types of 
reports and other methods of reporting can also be used with 5 
the PMAP system 401. 

Additional alternative embodiments will be apparent to 
those skilled in the art to which the present invention 
pertains without departing from its spirit and scope. For 
example, additional or alternative performance can be used 10 
or the definition of the performance measurements described 
can be modified. Accordingly, the scope of the present 
invention is described by the appended claims and is sup- 
ported by the foregoing description. 

What is claimed is: 15 

1. A method for comparing a service provided to a first 
local exchange carrier and a second local exchange carrier, 
comprising: 

collecting a first set of performance data associated with 

the service provided to the first local exchange carrier; 20 
normalizing the first set of performance data to determine 

a first performance measurement; 
collecting a second set of performance data associated 

with the service provided to the second local exchange 1S 

carrier; 

normalizing the second set of performance data to deter- 
mine a second performance measurement; and 

comparing the first performance measurement and the 
second performance measurement to compare the ser- 30 
vice provided to the first local exchange carrier with the 
service provide to the second local exchange carrier. 

2. The method of claim 1, wherein the first set of 
performance data and the second set of performance data are 
collected from process associated will first local exchange 35 
carrier. 

3. The method of claim 1, wherein the first performance 
measurement and the second performance measurement 
measure timeliness. 

4. The method of claim 1, wherein the first performance 40 
measurement and the second performance measurement 
measure accuracy. 

5. The method of claim 1, wherein the first performance 
measurement and the second performance measurement 
measure availability. 45 

6. A method for determining a performance measurement, 
comprising: 

collecting a first set of performance data associated with 

a first local exchange carrier; 
normalizing the first set of performance data to determine 

the performance measurement for the first local 

exchange carrier; and 
providing the performance measurement for the first local 

exchange carrier. 
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7. The method of claim 6, wherein providing the perfor- 
mance measurement for the first local exchange carrier 
comprise: 

providing the performance measurement for the first local 
exchange carrier via a network. 

8. The method of claim 6, further comprising: 
collecting a second set of performance data associated 

with a second local exchange carrier; and 
normalizing the second set of performance data to deter- 
mine the performance measurement for the second 
local exchange carrier. 

9. The method of claim 8, further comprising: 
comparing the performance measurement for the first 

local exchange carrier and the performance measure- 
ment for the second local exchange carrier. 

10. The method of claim 8, wherein the first local 
exchange carrier is an incumbent local exchange carrier and 
the second local exchange carrier is a competitive local 
exchange carrier. 

11. A method for collecting and analyzing performance 
data, comprising: 

collecting performance data for a first local exchange 
carrier from a process associated with the first local 
exchange carrier; 

normalizing the performance data for the first local 
exchange carrier to determine a performance measure- 
ment for the first local exchange carrier; 

collecting performance data for a second local exchange 
carrier from the process associated with the first local 
exchange carrier; 

normalizing the performance data for the second local 
exchange carrier to determine a performance measure- 
ment for a second local exchange carrier, and 

providing the performance measurement for the first local 
exchange carrier and the performance measurement for 
the second local exchange carrier. 

12. The method of claim 11, wherein the first local 
exchange carrier is an incumbent local exchange carrier and 
the second local exchange carrier is a competitive local 
exchange carrier. 

13. The method of claim 11, wherein the performance 
measurement for the first local exchange carrier and the 
performance measurement for the second local exchange 
carrier measure timeliness. 

14. The method of claim 11, wherein the performance 
measurement for the first local exchange carrier and the 
performance measurement for the second local exchange 
carrier measure accuracy. 

15. The method of claim 11, wherein the performance 
measurement for the first local exchange carrier and the 
performance measurement for the second local exchange 
carrier measure availability. 

***** 
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